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To determine the accuracy and usefulness of the National Emergency Department Over- 
crowding Study (NEDOCS) tool in an urban hospital ED in Australia by direct comparison 
with subjective assessment by senior ED staff. 

A sample of simultaneous subjective and objective data pairs were collected six times a 
day for a period of 3 weeks. All senior medical staff in the ED answered a brief question- 
naire along with the senior charge nurse for the ED. Simultaneously, the senior charge 
nurse also documented the total number of patients in the ED, the number of patients 
awaiting admission, the number of patients on ventilators, the longest time waited by an 
ED patient for ward bed, and the waiting time for the last patient from the Waiting Room 
placed on a trolley. The objective indicators were entered into a Web-based NEDOCS tool 
and transformed scores were compared with the averaged and transformed subjective 
scores for each sample time. Bland-Altmann and Kappa statistics were used to test the 
agreement between the objective and subjective measuring methods. 

The mean difference between the subjective and objective methods was small (3.5 [95% 
confidence interval -0.875-7.878]); however, the 95% limits of agreement was wide 
(-46.52-53.43). The Kappa statistic used to assess the extent of reproducibility between 
categorical variables was 0.31 (95% confidence interval 0.17-0.45). 

The present study suggests that NEDOCS method of processing the objective overcrowd- 
ing data does not accurately reflect the subjective assessment of the senior staff working 
at that time in the ED. This might be because the assumptions of the original NEDOCS 
study are flawed. 

Australian emergency department, emergency department, National Emergency Department 
Overcrowding Study tool, objective score, overcrowding. 
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Introduction 

ED overcrowding is a growing issue worldwide. Over 
the years various articles in the press, 1 ' 2 government 
reports and journals 3 ' 4 have highlighted this issue. It 
was first documented in Australia in a Sydney metro- 
politan hospital in late 1980. 5 Overcrowding of the ED 
has been demonstrated to have negative effect on 
patient care, 6 " 8 staff satisfaction and health. 9 ' 10 

Early warning of overcrowding would allow an orga- 
nizational response to prevent any potential adverse 
outcomes. Strategies could include calling in of extra 
staff, 11 ' 12 changing management strategies, 13 " 17 hospital 
ambulance bypass and notification of neighbouring 
hospitals. There is need for a tool, which can rapidly 
and accurately forewarn of overcrowding. 18 

Arguably the most accurate indicators of overcrowd- 
ing are the adverse events and complaints, 19 but there 
is a delay in notification and correlation. They are not 
helpful as an early warning system. 

The computerization of hospital admissions opens 
the possibility of a computer generated, automatic and 
objective warning system. 20 

Weiss and colleagues have explored the relationship 
of the computer-generated indicators of overcrowding 
with the subjective assessment by senior staff at 
the time of sampling. This model is called National 
Emergency Department Overcrowding Study 
(NEDOCS) tool. 21 

The NEDOCS tool was produced to 'quantitatively 
describe the staffs sense of overcrowding' in various 
sized academic ED in the USA. It was recognized that 
there is currently no gold standard measure of ED over 
crowding, so a qualitative measure of overcrowding 
was devised and described by same team. 22 This qual- 
itative measure was transformed into the subjective 
score. The NEDOCS Tool is a Web-based calculator, 
which converts a simple dataset into a number (objec- 
tive score) said to correlate accurately with degree of 
overcrowding as perceived by senior staff working at 
the time (subjective score). The NEDOCS tool has been 
developed for the use in clinical setting. It began as a 
23 site-sampling questionnaire reflecting the subjective 
assessment of ED physicians, 22 which was then statis- 
tically reduced using mixed effect linear regression to 
develop a 5-questionnaire model. The validity of the 
reduced model was tested using a bootstrap technique 
and was found to reflect the full model with 88% accu- 
racy. The NEDOCS tool was also found to correlate well 
with the number of patients who left without being 
seen 23 in further trials by the inception team. 



There has been an interest in implementing the 
NEDOCS tool in Australia. 

The objective of the present study was to assess the 
usefulness and accuracy of the Web-based model of the 
NEDOCS tool in an Australian ED compared with sub- 
jective clinician assessment. 

Method 

Ipswich Hospital is a 317-bed, non-tertiary urban hos- 
pital, located in Ipswich, Qld, Australia with an annual 
ED attendance of over 42 000. The ethics committee for 
Ispwich Hospital approved the study. Patient consent 
was not required. 

The present prospective pilot study covered a sample 
period of 3 weeks from 20 December 2004 to 10 January 
2005. Samples were collected at four hourly intervals 
(1.00, 5.00, 9.00, 13.00, 17.00 and 21.00 hours). The 
method duplicated the original paper by Weiss, 21 except 
that sample times were not randomized and the study 
was undertaken in a single hospital. The samples were 
collected six times a day although the study by Weiss 
covered each sample time twice only in the 3 week 
period. 

The composite outcome variable score (subjective 
score) was calculated as follows. On each shift between 
one and four senior medical officers present completed 
a short survey, consisting of two Likert type scales 
(Appendices 1,2) rating the 'Degree of overcrowding' 
and 'Feeling of being rushed'. No instructions or defini- 
tions other than the terms presented were given. The 
senior charge nurse of the ED was asked independently, 
but simultaneously to rate the degree of overcrowding 
using the same scale. The senior charge nurse was not 
asked to rate 'Feeling of being rushed'. The even Likert 
scale was used so that a break point for 'not over- 
crowded' versus 'overcrowded' fell between 3 and 4. 
Each timed and dated survey form was placed in a 
sealed box in ED. In keeping with the Weiss 21 study, for 
ease of interpretation the six-point scale was converted 
to a scale ranging from 0 to 200 (0 = not busy, 
40 = busy, 80 = extremely busy but not overcrowded, 
120 = overcrowded, 160 = severely overcrowded, 200 = 
dangerously overcrowded). 

The survey results of 'Degree of overcrowding' and 
'Feeling of being rushed' were averaged for all senior 
physicians and charge nurse for the ED working at the 
time of data collection. These averaged results formed 
the 'outcome variable' or 'composite score'. This score 
comprised the subjective rating of each paired sample. 
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Interrater agreement was tested retrospectively on 
the composite outcome data collected in the first week. 
The intraclass correlation coefficient was calculated for 
the responses to the 'feeling rushed' question asked 
from the doctors and then again for the 'overcrowding' 
question asked from the doctors and the charge nurse. 

The objective rating in each sample pair was calcu- 
lated by the Web-based NEDOCS Calculating Tool 24 
using the following objective measurements: 

1. Total number of patients in ED occupying beds. 
(Patients in waiting area were not included.) 

2. Total number of patients on ventilators. 

3. Total number of patients awaiting admission. 

4. Waiting time for the last patient called in from wait- 
ing room. (Patient occupying ED bed, waiting to be 
seen by the physician). 

5. Longest time patient waiting for admission. 

6. Number of beds in ED. 

7. Number of total beds (occupied and vacant) in 
hospital. 

The first five variable objective measures were collected 
simultaneously and prospectively using a prepared 
data sheet subsequently placed in a sealed box in ED. 
The ED attendance screen was printed at the same time 
to check for the accuracy of data collection. 

The NEDOCS Calculating Tool amalgamated the 
seven objective data points for each time of data collec- 
tion and rated the data for that sample time on follow- 
ing scale: 0-20 = not busy, 20-60 = busy, 60-100 
extremely busy but not overcrowded, 100- 
140 = overcrowded, 140-180 severely overcrowded, 
180-200 = dangerously overcrowded. 

Being a pilot study we aimed to duplicate Weiss's 
sample method for the same time duration as in original 
study, 21 rather than calculate a new sample size. How- 
ever, we used different statistical analysis as we had 
concerns about the product moment correlation (r) coef- 
ficient used by Weiss. The valid use of correlation coef- 
ficient requires a null hypothesis stating that there is 
no relationship between the two statistics. This would 
not be the case as both subjective and objective scores 
are being collected at the same time to assess the same 
thing, that is, 'overcrowding'. In our pilot study the 
Bland- Altaian 25 ' 26 plot was chosen for statistical analy- 
sis, to compensate for this error. Differences between 
measurements within each pair were plotted on the Y- 
axis. The averages of the two measurements were plot- 
ted on the X-axis. The average was used, as this was 
the closest estimate of the true result in the absence of 
a gold standard for 'over-crowding'. The mean differ- 
ence of all pairs was also calculated reflecting the 



'estimated bias' or the systematic difference between 
the two methods. 

Kappa statistics were used to analyse agreement 
between subjective and objective measures as well as 
assessment of interrater agreement on subjective 
responses. 

Results 

During the 3 week study period 2293 patients attended 
the ED, an average of 109 patients per day. For the 
entire 2004 calendar year, the Ipswich Hospital ED aver- 
aged 115 patients per day. 

One hundred and twenty-eight sample times were 
described by two scores (subjective and NEDOCS) 
over 3 weeks. The datasets for the 3 week period were 
complete. 

Figure 1 shows a scatter diagram of transformed 
subjective and objective (NEDOCS) scores. Figure 2 
shows that the mean difference between the methods of 
measurement is small (3.47, 95% confidence interval 
[CI] -0.875-7.878). Figure 2 also illustrates that the 
range of the 95% limits of agreement are wide (-46.52- 
53.43). 

The Bland- Altman plot shows that in 5% of the 
measurements, one method of measurement is likely to 
be more than 100-point raw score or 2.5 categories 
beyond the other. In the present study 16 pairs (12.5% 
[95% CI 7.8-19.3]) had scores more than 40 points (one 
whole category) apart and 49 pairs (38% [95% CI 30- 
46.9]) had scores in adjacent categories. The kappa 
value of 0.31 (95% CI 0.17-0.45) suggests poor agree- 
ment between the categorical scores. 

Interrater agreement of the subjective responses used 
in the composite outcome score was high. The intraclass 
correlation coefficient was 0.87 (95% CI 0.71-0.95) for 
the question on 'feeling rushed'. For the 'degree of over- 
crowding' the intraclass correlation coefficient was 0.94 
(95% CI 0.82-0.97). 

Discussion 

The present study demonstrates that the NEDOCS tool 
correlated poorly with subjective clinician assessment 
of overcrowding in our ED. The poor correlation is very 
obvious looking at Figure 2. The 95% limits of agree- 
ment are wide (more than two categories apart) making 
the NEDOCS tool imprecise when used in clinical set- 
ting. Such imprecision is unacceptable if one is 
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Figure 1. Scatter diagram comparison of both raw scores. (•) National Emergency Department Overcrowding Study (NEDOCS) and 
composite score. 
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Figure 2. Bland- Altman plot. 



considering up scaling hospital responses based purely 
on the NEDOCS score. In addition, Figure 1 shows us 
that there is no specific relationship between the NEDOCS 
and subjective scores. These findings indicate that the 
NEDOCS tool is not ready to be used in Australia. 

Our findings might differ from Weiss's original study 
for several reasons. The NEDOCS tool was developed 



in the USA and might not be applicable in a regional 
Australian hospital with different patient flow and 
staffing profile. 

The difference in findings may be partly explained 
by our decision to use different statistical methodology 
to Weiss and to interpretation of data results. Weiss 
used product moment correlation (r) coefficient for 
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statistical analysis in their original study, and the basis 
of its calculation may be questioned. They concluded 
that the NEDOCS score accurately reflects staffs sense 
of 'overcrowding' quoting R 2 0.49 for the full model. The 
NEDOCS tool reflects the full model with 88% accuracy 
so it is even less accurate. 

Weiss quoted a high interrater agreement (correlation 
coefficient 0.86-0.93) between physicians' and nurses' 
Likert results and between each result and the average 
as a justification for their use as composite outcome 
score. The Likert like scale used for subjective scores 
were developed from a preliminary survey of ED 
directors 22 who identified lack of ED beds, ED closure 
and ambulance diversion, patients managed in hall- 
ways, full waiting rooms, feeling 'rushed' and a wait of 
greater than 1 h to see a physician as evidence of ED 
overcrowding. 

The statistical analysis of the data where one mea- 
surement method (subjective score) is compared with 
another measurement method (objective score) can be 
fraught with danger. When there is no gold standard to 
compare the new measuring tool, the statistical analysis 
must take into consideration the possible error of each 
method. For that reason we chose the Bland-Altman 
plot and subsequently demonstrated poor agreement. 

A review of the literature highlights the difficulty in 
defining overcrowding. Hwang et al. 27 reviewed the lit- 
erature over 26 years and found a total of 230 articles 
relating to overcrowding in the ED including original 
articles, reviews and editorials. Only 23 of these had 
explicit definitions of overcrowding. There was sub- 
stantial variation in the definition of 'overcrowding in 
the ED' within these articles. The definitions focused on 
specific issues like waiting time, hospital-related but 
non-ED factors or factors external to the hospital such 
as ambulance diversions to describe overcrowding. 

There is little literature documenting quantitative 
measurements of overcrowding in the ED. 20 ' 21 Bernstein 
etal. developed the 'Emergency Department Work 
Index' (EDWIN index) which included: (i) number of 
patients in triage category; (ii) triage category; (iii) 
number of attending physicians on duty; (iv) number 
of treatment bays; and (v) number of admitted patients 
in ED. The limitation of EDWIN index is that it is not 
a computer-based model. It does not take into account 
various other issues such as the role of a resident in 
ED and also does not have any input from nursing 
staff. 

Weiss etal? 1,22 recognized this deficit in knowledge 
and undertook the difficult but important task of devel- 
oping a computer-based objective model. In its current 



form, this computer-based model NEDOCS tool has cor- 
related poorly with staff's sense of overcrowding in our 
hospital. For the moment, subjective assessment seems 
to be the best method of assessing overcrowding until 
we find a better tool or refine the NEDOCS scoring 
method. 

The NEDOCS scoring method is easy, simple and 
quick to use and could become a more important and 
useful tool with further refinement. A refined NEDOCS 
tool may provide us with the definitions and scoring 
tool we need. As Weiss suggests, measurements that 
include case mix within ED, triage category and skill 
mix of staff may improve agreement between NEDOCS 
score and staff's perception of 'overcrowding'. 

There are several limitations to the present study. At 
no time during the study was the department severely 
or dangerously overcrowded. Correlation might be 
better or worse at these extreme times. The study was 
conducted at only one urban hospital ED with mixed 
children and adult presentations and might not be appli- 
cable to other systems or private ED. The biggest lim- 
itation of the present study, however, is the lack of a 
'gold standard' by which to define ED overcrowding. 
We chose to use subjective physician/nurse assessment 
as the gold standard by which to measure the NEDOCS 
tool. It is possible that the NEDOCS tool might actually 
be very good, and that staff subjective assessment is 
actually poor. 

Conclusion 

The NEDOCS tool was not valid in our setting and has 
been inconsistent in reflecting staff's sense of 'over- 
crowding' for which it was primarily designed. 
NEDOCS is still a potentially important tool worthy of 
refinement. 
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Appendix I 

Survey form for ED physician for NEDOCS tool 

Date: 

Please circle the time. 

Time: 1 AM 5 AM 9 AM 1 PM 5 PM 9 PM 
Please circle the opinion on 'Degree of Overcrowding' 



1 


2 3 4 5 6 


1 


= not busy 


2 


= busy 


3 


= extremely busy but not overcrowded 


4 


= overcrowded 


5 


= severely overcrowded 


6 


= dangerously overcrowded 



Please circle the opinion on 'Feeling rushed' in ED 
1 2 3 4 5 6 
1 = not rushed 
6 = rushed 
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After completing the survey form, please place it into 
sealed box provided in Emergency Department. 
ED, emergency department; NEDOCS, National Emer- 
gency Department Over Crowding Study. 

Appendix II 

Survey form for the charge nurse in ED for 
NEDOCS tool 

Date: 

Please circle the time. 

Time: 1 AM 5 AM 9 AM 1 PM 5 PM 9 PM 



Please circle the 'Degree of Overcrowding'. 
1 2 3 4 5 6 

1 = not busy 

2 = busy 

3 = extremely busy 

4 = overcrowded 

5 = severely overcrowded 

6 = dangerously overcrowded 

After completing the survey form, please place it in the 
sealed box provided in Emergency Department. 
ED, emergency department; NEDOCS, National 
Emergency Department Over Crowding Study. 



288 



© 2006 The Authors 

Journal compilation © 2006 Australasian College for Emergency Medicine and Australasian Society for Emergency Medicine 



